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IMAGE SENSOR WITH MULTIPLE RESOLUTION AND COMPRESSION CAPABILITY 

Background of the Invention 

The present invention relate* generally t0 an apparatm md method for x 
mterpolating and compressing image data and. more particularly, to an apparatus and 
method for sensing color sub-samp..d image data and thereafter interposing for 
non-color sub.san.ple4 image data tha, substantia,* reduces coior fringing and 
preserves detaii while allowing selection of the overall quantity of image data. 

Electronic imaging cameras for recording sti.1 images are well known in the ar, 
Such cameras can recnrd a plurality of still images on a single magnetic disk or tape 
» ether analog or digital forma, for subsequent payback on an output device, such 
as any weU-known cathode ray tube (CRT) viewing device for example Printers 
may also be utilized with suc h cameras in a well-known manner to provide hard 
copies of the recorded images. 

Such electronic imaging still earner often utilize two-dimensional image 
senstng arrays such as charge coupled devices (CCDs) which integrate incident 
scene light over a predetermined time to provide an e,ectro„ic information signal 
corresponding to Ute scene light intenshy i ncidrat on me ^ ^ ^ 

dtmenstonal image sensing arrays comprise a predetermined number of discrete 
■mage sensing elements or pi « ls ^ „ . ^ fa ^ ^ 

■mage sensing element responds to incident Ulummation to provide an electronic 
^formation signal corresponding to the intensity of the incident of illumination 
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In order ,o record color images, *. farfd-t il.umination to 4. , wa dimensiona , 
■mage — ta. array is fiItered so differOT , image ^.^ denients ^ 
d,ffe re „, co.ored Ulumination. a. fllters „ ^ jn we „ ^ ^ ^ 
the face of an image sensing ^ ^ , repeating ^ rf ^ ^ ^ ^ 
stnpes. Alternative*. i„ dividu a, image sensing events or pixels ^ ^ ^ 
may be fhered in a repeating pattern of red, green. b.ue, and green fi.te in . ^ 
by-two checkerboard, as is «.„ known in «,,. m . Since ^ ^ ^ ^ 
can only detect one color of illumination, the co.or information for thelother co.ots 
no, detected by tha, image sensing Cement must be filled i„. Filling q missjng 
color information is generally accomplished by interposing the detected image data 
for each color to determine color va.„es for all the colors for each iriage sensing 
element. 

Conventional types of interpolation, however, can provide ijnages with 
objectionable color aliasing artifacts such as "color fringes" near sharp; edges. The 
conventional approach to solve this problem is to prevent the color fijinges at the 
expense of image sharpness by blurring, otherwise known as anti-aliasing, the 
Picture so that the edges are not sharp enough to create a color fringe. iBlurring the 
image in this manner, however, has obvious disadvantages, resulting irl a reduction 
in resolution and a so-called "fuzzy" picture. j 

The missing color interpolation also increases the number of pixels tjy a factor of 
three from the raw image data sensed by the CCD. Storing the image! data locally 
then oecomes a potential problem. The increased number of pixels meahs that a file 
size for storing the image, on local magnetic media for example,! leads to a 
requirement for increased storage capacity. 
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1*. pixe l5 „ a CCD can be rec*ngular to ^ ^ ^ 

finer arrays are emp lo yed. Thus, outputting ilMge ^ ^ ^ ^ 
substantially souare pixels ^ , hc image sinM ^ 

devtces such as printers ana CRT's reouire square pixels , it „ ^ ^ ^ 

^ ^ ** ^ "* — " ** .o outputting image 
A result is to get the desired pixels geometry. 

Summary 

The aforementi one d and other objects arc achieved by the invention which 
provdes, i„ one aspect an electronic StiU camera and a method associated therewith 
for .mage processing to form a processed image with reduced coior artifacts where 
an .mage is represented by a plurality of image signais in three Cor, The method 
« useru, with a. Cectionic stil, camera, and comprises the steps of separating the 
Pluraiity of image signais into a first color piane, a second coior plane, and a third 
coior piane. For exampie. mar* convention* CCDs use coiored stirpes formed onto 
»e CCD ,„ filter coiors, thereby making individual pixels associated with only one 
color. These individual pixeis are broken into color p ( anes. Therefore, if colored 
«npes of red, green, and b.ue (RGB) are attached ,„ the CCD, then the pixels 
associafcd with each individual Cor are bmken into a red coior p,ane, a green color 
plane, and a blue color plane. 

Since these dors have now been separated, voids now exist where the previous 
colors used to be. That is, in the red color plane, the pixel locations where green and 
blue had ,„ have existed previous.y are now left void. Therefore, the ne x, step is ,„ 
■merjec. interpoiated image sigr^s in each of the color planes tha, correspond ,„ the 
■mage s igMl locations !e ft void by the separation step. This „ ow forrts . ^ fm 
-h color location in the image. Previously, a single pix e, .ocation wou,d contain 
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only information on one color, red for example. Now. with ,he interpo.ated dau,, 
each pixel location has RGB information associated therewith. 

Nex^ two difference signals are created at each pixel location that represent the 
differences between one channel and a second, ami the firs, channel and the third 
within the ROB triplet. In the previous example using RGB, those difference signals 
can be any combination of the red, the green, or the blue, as long as from those three 
colors, two different difference signals are created. In the preferred embodiment, R- 
G and B-G are used. 

The difference signals are then filtered using a median filter which substantially 
reduces or removes color artifacts. The image is subsequently reconstructed from 
the original image signals and the filtered difference signals to form a processed 
image which has substantially preserved detail and reduced color artifacts. The 
improved RGB triplets thus formed from linearly interpolated RGB triplets have the 
same pixel geometry as the original single color pixels. In the case of rectangular 
Pixel geometry, some method of producing pixels of square geometry must be 
employed for use with any display device that renders images on a ras,er of square 
geometry. 

In the case of rectangular pixels of aspect ratio height twice the width, the 
invention provides dual resolution modes for easily producing square pixels for 
display. I„ one resolution mode, neighboring pairs of pixels are averaged in the 
direction normal to the major axis of the rectangular pixels within each color plane 
such that one half as many square pixel RGB triplets are formed as^ there were 
original single color rectangular pixels, while in the second resolution mode new 
RGB triplets are interjected by interpolation in the direction parallel t o the major 
axis of the rectangular pixels such that twice as many square pixel RGlji triplets are 
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formed as there were original single color rectangular pixels. In the case of 
rectangular pixels of aspect ratio other than height twice width, multi resolution 
mode can be supported with more complicated down sampling and up sampling 
schemes. 

Finally, if the image data is to be stored, then compression becomes an essential 
characteristic. In the invention, the stored image signals are compressed versions of 
the two filtered difference signals and a compressed version of the single color 
signal common to both difference signals reconstructed from the filtered difference 
signals and the original sampled pixel values. In the preferred embodiment, 
compressed, filtered R-G and B-G are used as well as a compressed version of the G 
signal formed from the filtered R-G and B-G and the original RGB samples. The 
difference signals are highly compressible while the G signal formed from the 
filtered R-G and B-G and the original RGB samples provides the complete detail 
signal originally sensed, and the overall image size can be drastically compressed. 
Upon decompression of the compressed image data, pixels of square geometry can 
be reconstructed in either of the dual resolution modes. The aforementioned and 
other aspects of the invention are evident in the drawings and in the description that 
follows. 

Brief Description of the Drawings 

The foregoing and other objects of this invention, the various features thereof, 
as well as the invention itself, may be more fully understood from the following 
description, when read together with the accompanying drawings in which: 

Figs. 1A and IB show, respectively, pixel locations as found on a CCD, and a 
representative example; 
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Figs. 2A and 2B show the pixel locations of the example of Figs.* 1A and IB, 
respectively, after linear interpolation is performed; 

Fig. 3 shows the representative example of Figs. IB and 2B during performance 
of median filtering; 

Fig. 4 shows the result of the median filter on the example of Fig. 3; ! 
Fig. 5 shows reconstruction of pixels of Fig. 3; and 

Figs. 6A, 6B, and 6C show, respectively, the pixel values reconstructed after 
median filtration of the color difference signals, the pixel values after averaging in 
the direction normal to the major axis of the rectangular pixels, and the pixel values 
after interpolation along the major axis of the rectangular pixels. 

Detailed Description 

i 

i 

While the present invention retains utility within a wide variety of photographic 
devices, such as video cameras for example, and may be embodied in several 
different forms, it is advantageously employed in connection with an electronic still 
camera. Though this is the form of the preferred embodiment and will be described 
as such, this embodiment should be considered illustrative and not restrictive. 

As previously discussed, electronic imaging cameras conventionally record 
color images by using an image sensing array comprising a predeterrnined number of 
discrete image sensing elements or pixels arranged in a two-dimensidnal array in 
which the image sensing elements respond to incident illumination td provide an 
electronic information signal corresponding to the intensity of the incident 
illumination. Such an image sensing arrays may be a CCD of the frame transfer 
type. It is well known to sense color images using a single two-dimensional CCD 
array by filtering the illumination incident to the image sensing array so that 
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different groups of the image sensing elements arranged in well-known patterns 
across the image sensing array receive different wavelengths for colored 
illumination. Thus, each color of illumination is sampled by each group of image 
sensing elements, and thereafter interpolated to provide color values corresponding 
to the other groups of image sensing elements. The full color image is therefore 
estimated or interpolated between the different groups of image sensing elements or 
pixels to fill in all colors for each image sensing element or pixel. 

Conventional types of interpolation provide images with objectionable aliasing 
artifacts such as color fringes near sharp edges. An example of how a sharp edge in 
a sample to be recorded can create color fringes, when the image of the subject is 
reconstructed using conventional interpolation methods, will be discussed herein. 

As previously described, a typical CCD arrangement includes color filter stripes 
thereon such that individual stripes of pixels measure an intensity of light for only a 
single color. Figure 1A shows a typical arrangement of a CCD using color stripes, 
where individual pixels are marked by R, G, or B representing red, green, and blue 
colors. Numerical designations associated with the several colors, i.e., R,. G,. and 
B|, together are referred to a triplet. 

In order to illustrate the invention, an example is shown for each step. The 
example is of an image having a sharp contrast such as a sharp white-to-black 
transition. The example begins in Fig. IB where a light intensity level is shown 
graphically for a white-to-black transition in an image. For the graph, the abscissa 
is , , pixel number across the CCD and the ordinate is by signal level, often 
measured in volts. Keys to Fig. IB is as follows: the solid line represents the edge 
transition in the original continuous image of the scene before sampling, the circle 
represents an original red sample, the triangle represents an original green sample, 
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and the square represents an original blue sample. It can be seen from the 
illustration that for the original continuous image the falloff during the! transition is 
substantially vertical. This indicates a sharp transition. A slope in the falloff would 
be indicative of a more gradual, or less sharp, transition. 

Fig. 2A again shows a pixel layout of the pixels of Fig. 1A, but now the pixels 
have gone through the step of interpolation. In the preferred embodiment, this is a 
linear interpolation, as will be later hereinafter described. The convention used is 
that a capital letter represents original raw data and a lower case letter represents 
interpolated data. Therefore, the colors interpolated between R, and R 2 are shown as 
rr and r,». Using linear interpolation, r,' contains 2/3 of the intensity of R„ and 1/3 
of the intensity of R 2 . Likewise, rf contains 1/3 the intensity of R, land 2/3 the 
intensity of R 2 . By linearly interpolating values throughout for red, gree^ and blue, 
the colors can be broken into three color planes such that there now exist* triplets for 
each individual pixel location, where previously there only existed rawidata in one 
color for each location. 

Continuing the example, Fig. 2B shows the step of linear interpolation on the 
white-to-black transition shown in Fig. IB. As shown in Fig. IB, the lines indicate 
paths of the linear interpolation where the circles, triangles, and squares represent 
original red, green, and blue samples and the x's represent interpolated values. 
Figure 2B illustrates how the interpolation tends to blur the sharp transition 
previously seen in the image by interpolating where the dashed line represents red, 
the dotted line represents green, and the solid line represents blue. The ,slope of the 
line is indicative of a less sharp transition, thus blurring the image. Ifig. 2B also 
illustrates how the interpolation tends to create color fringe artifacts. ln, the original 
white-to-black transition the red, green, and blue intensity transitioned , at the same 
spatial location while it can be seen in Fig. 2B that the red signal after interpolation 
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initiates transition before the green which initiates transition before the blue. An 
image reconstructed from the data represented in Fig. 2B would show a pronounced 
white-to-cyan-to-purple-to-blue-to-black color fringe. 

A next step is to create two difference signals at each location. The color fringe 
artifact at a white-to-black transition in the scene followed by interpolation within 
color planes would appear as a sudden rise in a color difference signal followed by a 
corresponding sudden fall or as a sudden fall followed by a sudden rise. It is this 
rapid increase and decrease in the difference between the colors which is a 
characteristic of objectionable color fringing, and is not simply a sudden rise in the 
difference between colors, which is indicative of a change from one color to a 
different color. Thus, it is unlikely that a real scene would result in the creation of 
such a color spike, and it is not desirable to create such a color spike as a result of a 
method of interpolation chosen. 

Fig. 3 illustrates the aforementioned color spikes, where in the preferred 
embodiment, the difference signals are performed by subtracting the intensity of 
green from the intensity of red and subtracting the intensity of green from the 
intensity of blue, that is and B-G. One skilled in the art will realize that the 
actual choice of colors that determine the difference signals is somewhat arbitrary. 
Consistency after the choice is made is of primary importance. Therefore, other 
color difference signals can be chosen without detriment to the invention. Fig. 3(i) 
graphically illustrates the results of the difference signal for R-G, for the signal 
shown in Figure 2B. Fig. 3(iii) graphically illustrates B-G. 

Various methods can now be employed to remove the color aberration. In the 
preferred embodiment, a median filter is employed, such as that shown in commonly 
assigned patents 4,663,655, issued May 5, 1987, 4,774,565, issued September 27, 
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1988, and 4,724,395, issued February 8, 1988, all to William T. Freeing which are 
each incorporated herein by reference. Simply anted, the median filter takes a series 
of pixels, such as those shown in Fig. 3(ii), and replaces the pixel value at the center 
of the filter region with the median value of all the pixels within the region. Though 
various numbers of pixels can be used, in the preferred embodiment, the median 
filter uses nine pixels and returns the median value of the nine pixels. Therefore, as 
the filter is worked horizontally across the signal, it can be seen that unless the spike 
is longer than one half the pixel length, the spike in the signal will b* essentially 
eliminated. 

i 

Figure 4 illustrates the results of passing the median filter across the difference 
signals, where 4(i) corresponds to the R-G difference signal shown in 3(i), and 4(ii) 
corresponds to the B-G difference signal shown in 3(iii), where these ! signals can 
now be referred to as (R-G)' and (B-G)', respectively. The differences are now flat 
and the reconstructed colors will now remain substantially constant wi«h respect to 
each other before, during and after the color transition. Reference is again made to 
Figure IB as an example of this. 

A next step is shown in Figure 5, where the pixels are reconstructed from 
original raw RGB data and filtered (R-G)' and (B-G)' difference signals.! The colors 
are restored using the following relationships: 

R@R=R 1 

i 

R@G = G + (R-G)' 

R@B - B - (B-G)' + (R-G)' 

i 

i 

G@R = R-(R-G)' 

G@G = G ! 
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G@B = B-(B-G)' 
B@R = R - (R-G)* + (B-G)' 
B@G = G + (B-G)' 
B@B = B 

5 These relationships illustrate the properties of preserving the original sampled 

values and extracting detail at every pixel regardless of its original color. The detail 
extracting property is accomplished by removing the sudden spikes in the color 
difference signals which effectively produce local color sameness, sq in regions of 
sharp edge transitions, when the original sampled pixels values change in response 
10 to edge transitions, the reconstructed missing color values are made to follow the 
original values. This property injects the detail information into all three channels. 
Figure 5 illustrates the reconstruction, where now instead of having diagonal lines 
showing the transition between colors, illustrating loss of sharpness and color 
fringing, now the circle representing red, the triangle representing green, and the 
square representing blue at the transition points are located in a substantially 
identical location such that the sharpness is retained and color fringes are avoided. 

In order to display the resulting image on most commercially available CRTs and 
most commercially available printers, a square pixel must now be attained. Since 
many CCDs, and specifically that of the preferred embodiment, have rectangular 
pixels, generally having a 2 to 1 aspect ratio, a way of solving this problem is to 
average two neighboring pixels, R, and r,', for example, to form a single square 
pixel at that location. Fig. 6A shows pixels values from an RGB striped sensor with 
2:1 aspect ratio rectangular pixels after reconstruction from original sampled color 
values and median filtered color difference signals. Figure 6B shows pixel values on 
a square grid after averaging neighboring values. This would take an image 
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resolution that has one million pixels, for example, and create an image thafi, 
displayable to a user having 500,000 three color pixels. An alternative method of 
creating square pixels is optionally used in the preferred embodiment. Rather than 
averaging horizontally, the pixels are interpolated vertically, as is shown in Fig. 6C. 

In Fig. 6C(i), an original line of RGB data is shown, and in Fig. 6C(iii) a second 
line of RGB data is shown, separated by Fig. 6C(ii) which is an interpolated line of 
RGB data. The interpolation between these lines can be as simple as nearest 
neighbor interpolation, which, in essence, duplicates a previous line to create a new 
line. The preferred methods include linear interpolation, bi-cubic convolution 
interpolation, or frequency domain interpolation, such as Fourier, DCT, wavelet, et 
cetera. 1 

The interpolated line of Fig. 6C(ii) creates additional pixels available for display. 
Thus, in the one million pixel example, two million three color pixels would now be 
available. That is, if the image was 1600 x 6 00 x 3 where the three represents the 
three color planes, interpolation would result in 1600 x 1 200 x 3 available pixels. 

In contrast to the prior art method of beginning and ending with square pixels, 
the method of the invention does not, per se, result in square pixels. The pixel 
geometry has actually not changed. The method works since output devices do not 
actually know the geometry of the pixels coming into the device, but instead simply 
displays according to an electronic signal. Therefore, rectangular pixels are 
displayed either separated by an interpolated line which carries color information 
that makes the transition between the rectangular pixels or effectively formed into 
squares by averaging. This serves to eliminate the distortion that would otherwise 
result and simulates having a square pixels geometry. 
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The resulting increase in pixels bodes well for output image quality but presents 
a difficulty for data storage. Increasing the image pixel count by twb requires an 
increase in storage capacity for an image by two. Therefore, compression is used to 
decrease image storage requirements. 

5 A characteristic that makes data highly compressible is redundancy. The actual 

amount of compression is determined by the compression method chosen. In the 
preferred embodiment, DCT compression is used which is described in commonly 
assigned patent application number 08/441,000 filed May 15, 1995 which is 
incorporated herein by reference. 

10 Signals that have a high degree of redundancy are the filtered difference signals 

as can be seen in Figure 4. The signals are compressed along with the reconstructed 
data of one color. The one color must be chosen such that the one color can be used 
to derive the remaining colors with the difference signals. In this case, green is used 
since both difference signals incorporate green. Therefore, the other colors can be 

15 derived by the following equations where the superscripted plus indicates that 
reconstructed data is incorporated therein: 

G = G + 

R = G + + (R-G)' 
B « G + + (B-G)' 

The detail is carried in the green G + signal which when the image is 
reconstructed is injected into the R and B signals by the above equations. 

In the preferred embodiment, visually lossless compression is used though 
lossless compression can be used without detriment to the invention. Visually 
lossless is preferred since higher compression ratios can be obtained, on the order of 
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0 :1 , whiie by defi^on changes ,o the reconstructed image „ no , ^ ^ ^ 
human eye. Furihermore, i„ ^ ^ ^ 

values with squarc geon , e0y jn ejIher rf ^ resoiut . ons , s 

expression as ^ i„ y ^ ^ ^ 

08/441,000. 

The invention may be embodied in other specific forms ^ ^ 
4e sp,„, „ r essentia, characteristics the^f. Th. present embodiment are 
therefore. «o be considered in a., re,pe«s as iHusfcative and no, reactive, the scope' 
of the invention being indicated by the appended claims raher than by tAe foregoing 
description, and a.! changes which come within the meaning and range of 
equivalency of the claims are therefor, intended to be embraced therein 
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Claims 

The embodiments of the invention in which an exclusive property or privilege is 
claimed are defined as follows: 

1 . A method of processing an image stored in an electronic still camera in a first 
resolution having a first pixel geometry and the electronic stijl camera is in 
communication with a display device having a second pixel geometry, the 
method comprising: 

5 processing the image to alter the first pixel geometry to form a processed 

image having the second pixel geometry; and 

transmitting the processed image to the display device for display thereon; 

2. The method according to claim 1 wherein said step of processing further 
comprises the steps of: 

separating the plurality of image signals into a first color plane, a second 
color plane and a third color plane; 

5 interjecting interpolated image signals into each color plane corresponding to 

image signal locations left void by said separation step to form a 
triplet for each location in the image; 

creating a first difference signal and a second difference signal representing a 
difference between two or more image signals for each triplet; 

10 filtering the first and second difference signals to create filtered difference 

signals; and 

reconstructing the image from the plurality of image signals and said filtered 
difference signals to form the processed image having the second 
pixel geometry. 

3. A method according to claim 2 wherein said step of reconstructing the image 
comprises generating pixel values according to the following criteria: 
R@R=R; R@G=G+(R-G)'; R@B=B-(B-G)'+(R-G)' 

where (B-G)' and (R-G)' are the first and second difference signals, 
5 respectively. 
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5. 



7. 



8. 



10. 



11. 



12. 



13. 



The method according to claim 2 further comprising the step of reducing a 
file size of the processed image for storage by compressing said difference 
signals and one color common to the difference signals. 1 

The method according to claim 1 further comprising the steps of: 

receiving a resolution signal from the display device; and 

altering the image such that the processed image has a second resolution not 
equal to the first resolution. 

i 

The method according to claim 5 wherein said step of altering the image 
comprises the step of averaging pixels in the image horizontally to form the 
processed image. 

The method according to claim 4 wherein said step of altering the image 
comprises the step of interpolating pixels in the image vertically to form the 
processed image. ! 

The method according to claim 7 wherein said step of interpolating pixels in 
the image uses linear interpolation. 

The method according to claim 7 wherein said step of interpolating pixels in 
the image uses bi-cubic convolution interpolation 

The method according to claim 7 wherein said step of interpolating pixels in 
the image uses frequency domain interpolation 

The method according to claim 7 wherein said step of interpolating pixels in 
the image uses nearest neighbor interpolation. 

An electronic still camera having a sensor with rectangular pixels which 
captures image-bearing light to form an image thereon, said camera being 
connectable to a computer having a display with square pixels, said camera 
further comprising processor means for converting rectangular pixels in the 
image to produce square pixels such that the image can be displayed on the 
display of the computer. 

The electronic still camera according to claim 12 wherein said processing 
means is selectable to transmit the image to the computer in one 6f a plurality 
of resolutions. 
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The electronic still camera according to claim 12 wherein said processing 
means is selectable to transmit the image to the computer in a first resolution 
or a second resolution mode where the first resolution mode is less than the 
second resolution mode. 

The electronic still camera according to claim 12 wherein said processing 
means averages neighboring pairs of the rectangular pixels in the direction 
normal to a major axis of the rectangular pixels to form the first resolution 
mode. 

The electronic still camera according to claim 12 wherein half as many 
square pixel RGB triplets are formed as there were original single color 
rectangular pixels. 

The electronic still camera according to claim 12 wherein said processing 
means interjects interpolated pixels in the direction parallel to the major axis 
of the rectangular pixels to form the second resolution mode. 

The electronic still camera according to claim 12 wherein twice as many 
square pixel RGB triplets are formed as there were original single color 
rectangular pixels. 

The electronic still camera according to claim 12 wherein! the image is 
compressed prior to storage. 

A method of storing an image captured by an electronic imaging device 
comprising the steps of: 

separating the image into a first color plane, a second color plane and a third 
color plane; 

interjecting interpolated pixels into each color plane corresponding to image 
signal locations left void by the separation step to form a triplet for 
each location in the image to produce complete color planes; 

creating a first difference signal and a second difference signal representing a 
difference between two or more image signals for each triplet; 

compressing the first difference signal, the second difference signal and one 
of the complete color planes where the one of the complete color 
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22. 



planes is chosen as being a color common to the first land second 
difference signals. 

The method according to claim 20 further comprising the step of 
decompressing the first difference signal, the second difference signal and 
one of the complete color planes to form a decompressed image substantially 
similar to the image. 

The method according to claim 20 wherein said step of decompressing 
performs reconstruction according to the following formula where G is 
representative of the one of the complete color planes common! to the first 
and second difference signals: 

G = G + 

R = G* + (R-G)' ! 
B = G + + (B-G)' ! 

and where the superscripted plus indicates that reconstructed data is 
incorporated therein. ' 

23. The method according to claim 22 wherein the G + signal contains color detail 
which when reconstructed is injected into the R and B signals by the above 
equations. ' 
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